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A series of five reports is presented which describes 
the activities carried out by the Pennsylvania State University group 
engaged in research in coiputer-assisted instruction (CAI) in 
vocational-technical education. The reports cover the period January 
1968-June 1968 and deal with: 1) prior knowledge and individualized 
instruction; 2) nuaerical and verbal aptitude tests adainisterea at 
the CAI station; 3) an experimental procedure for course revision 
based on students* past perforaance; 4) the developaent of a 
geometric dictionary for the IBM 1500 computer system; and 5) a 
processor for multiple numeric entries. Also provided is a preview of 
activities for fiscal year 1969. Curriculum development activities 
are described as focusing on the developaent of comnunications skills 
for the vocational student and the creation of CAI physics material. 
Research efforts are listed as centering upon: 1) the development of 
a computer-based seguential intelligence test; 2) adaptation of the 
General Aptitude Test Battery to on-line presentation; and 3) 
replication of prior studies contrasting the effects of gradient and 
full-response feedback on immediate learning and retention. 
(Author/PB) 
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FOREWORD 



The reports on the following pages comprise the sixth semi-annual descrip- 
tion of the activities of the Penn State group engaged 1n research on computer- 
assisted Instruction 1n vocational-technical education. To the reader who has 
not seen the previous five reports, this one will seem Inexplicably uncoordi- 
nated because 1t reports on research and development carried out on two differ- 
ent computer configurations. In fact, the period between January 1, 1968 and 
June 30, 1968 has been a time of transition— 1n hardware, from the IBM 1410 to 
the IBM 1500, and 1n author language, from Coursewrlter I (Yorktown Heights 
version) to Coursewrlter II. Our efforts during the period have gone Into the 
completion of experimental studies begun on the typewriter terminal, the trans- 
lation of carefully selected course material from the old author language Into 
a new and more powerful medium for reaching learners, and the development on 
the part of the staff of new skills and new teaching strategies for the richer 
learning environment provided by the 1500 system. 

During the fiscal year 1969, we expect to expand our curriculum develop- 
ment efforts 1n communications skills for the vocational student. A simulated 
physics laboratory has been outlined and partially completed. Emphasis 1n this 
1nqu1ry-or1ented physics material 1s on the understanding of basic physical 
laws that are required by the post high school technical education student. In 
addition, 1n the curriculum development segment for the fiscal year 1969, we 
will expand upon the material from a pilot study 1n computer-assisted occcupa- 
tlonal information completed last year with funds provided by the Pennsylvania 
Department of Public Instruction. 

Major research efforts for 1968-69 are planned around 1) the development 
and evaluation of a computer-based sequential intelligence test, 2) adaptation 
and appraisal of the General Ap titude Test Battery to "on-line" presentation, 
3) replication of prior studies contrasting the effects of gradient and full - 
responso feedback on Immediate learning and on retention. 
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In addition to the major curriculum development and research studies, we 
expect to continue dissemination about potential applications of CAI in voca- 
tional education and about computer systems Innovations on the IBM 1500. 

Harold E. MUzel 

University Park, Pennsylvania 

July 1, 1968 
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PRIOR KNOWLEDGE AND INDIVIDUALIZED INSTRUCTION 
Bobby R. Brown and Terry A. Bahn 

One of the often cited advantages of computer-assisted Instruction (CAI) 
1s the capability 1t affords for altering the nature and flow of Instructional 
material presented to Individual students. With CAI 1t 1s possible to adapt 
Instruction to the needs of each Individual student as reflected by measures 
of the student's past performance, by the student's differential aptitudes, or 
by some combination of both. CAI presently provides a considerable degree of 
Individualization of Instruction by allowing each student to proceed at his 
own rate and by providing remedial branching within "main track" programs. 
This Individualization of Instruction Is achieved through the use of response 
sensitive programs which make little use of measures of Individual differ- 
ences or of the student's history prior to his beginning the program. 

There are, as Cronbach (1267) has pointed out, two ways of adapting 
Instruction to Individual differences: "1) provide remedial adjuncts to 
fixed 'main track' Instruction, and 2) teach different pupils by different 
methods," A search of the literature reveals a general absence of pre- 
scriptive statements concerning how to "teach different students by differ- 
ent methods." A comparison of Smallwood's assessment of the situation 1n 1562, 
with conclusions drawn by Oilman and Gargula five years later, 1s Indicative of 
the relative lack of progress 1n this area. 

At this point we must rely on our Intuition to convince our- 
selves that a system capable of making systematic changes 1n Its 
presentation of material and 1n Its Internal decision process 
must be a potentially better teaching device than one without 
these advantages. The ultimate answer lies at the bottom of a 
mass of as yet uncollected data. (Smallwood, 1962, p. 108.) 

The results of this study Indicate that 1f branching 1s to 
be used to advantage 1n computer-assisted Instruction, there must 
be a thorough Investigation of those situations where 1t facili- 
tates learning. Also, research needs to be Implemented to 
determine the crltarl* for branching decisions, (Gllman and 
Gargula, 1967.) 
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Cronbach (1967) has suggested that "we ought to take a differential 
variable we think promising and design alternative treatments to Interact with 
that variable." In line with this suggestion, and 1n an effort to achieve the 
as yet Idealized goal of "optimal learning" for each student, more research 1n 
CAI needs to be directed toward the accomplishment of three highly interrelated 
tasks*. 

1. The delineation of valid and potent Individual differences 
among students, on the basis of which differential treat- 
ments will be of consequence 1n affecting student perfor- 
mance. 

2. The design of a repertoire of Instructional strategies by 
which material can be presented. 

3. The formulation of models or decision rules for pairing 
student and strategy such that each student receives the 
Instruction best suited to him. 

This paper reports the results of an experiment designed to Investigate a 
possible technique for adapting to Individual differences, specifically for 
adapting Instructional methods to the extent of prior knowledge. 

One of the most obvious ways 1n which students differ 1s 1n the extent of 
their prior knowledge of the subject matter to be taught (Glaser, 1968). The 
extent of prior knowledge has receives a considerable amount of attention from 
experimenters, not, however, as an Individual difference variable but as a 
troublesome source of experimental error. In their efforts to control this 
source of experimental error, experimenters have employed pretests as screening 
devices, or have used esoteric or contrived subject matter about which 
seb.1ects are unlikely to have any prior knowledge (or Interest). An example 
(if one Ingenious and effective means of assuring the naivete of subjects 
appears 1n a study by Merrill and Stolurow (1966) 1n which students were 
taught an "Imaginary" science. 

While the experimenter who 1s not Interested in studying this variable 
is Justified and perfectly correct in controlling 1t in his experiment, extra- 
experimental applications of CAI make its control Impractical 1f not Impos- 
sible. So despite the fact that most experimenters have chosen to consider 
extent of prior knowledge a source of experimental error, either to be Ignored 
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or controlled, the fact, remains that, w1* . long-range routine use of CAI, 
provision will have to be made for teaching students who differ 1n extent of 
prior knowledge of the subject matter to be taught. Because of different 
past educational experiences and differential forgetting, students will likely 
vary not only In amount but also 1n specific areas of prior knowledge. 

Programs which have been developed using only naive students may not be 
at all satisfactory for use with students who have various amounts of prior 
knowledge or' the subject being taught. In evaluating a course with all naive 
subjects two measures are commonly employed: time on program and amount 
learned. These two are often combined Into a single derived score of the form, 
"amount learned/time ■ efficiency score." However, 1f one 1s concerned with 
achieving criterion performance, that 1s some fixed, predetermined level of 
proficiency, the only measure of concern should be "time to criterion." An 
efficiency score established with naive students may be totally misleading, 
and the program may be very Inefficient for teaching students with prior 
knowledge of the subject matter, unless provision 1s made for skipping past 
the material which the students have already mastered. In addition to being 
inefficient, the program may have detrimental effects; for example, students 
who are required to proceed frame by frame through material which they have 
already mastered may become bored and disinterested with the entire program. 

If a program Is to be responsive to the needs of students who have 
various amounts of prior knowledge, It should possess the following: 

1. a means of assessing each student's knowledge of each concept 
or sub-concept prior to Instruction; 

2. a means of skipping past material which the student has 
already mastered; 

3. a means of providing a rapid review of the material about 
which the student has some prior knowledge; 

4. a means of providing Instruction on the material for which 
the student has little or no prior knowledge. 

An effort was made to provide these four capabilities 1n a CAI program. 

Three separate techniques were employed in an effort to prepare a 
program which would be adaptive to the needs of students possessing various 
amounts of prior knowledge. The reasoning behind the selection of each of 
these techniques 1s covered in the next section. 

ERIC 



4 



Rationale 

In the research literature on size of step as a program variable, the 
most consistent finding has been that students proceed more rapidly through 
large step programs. (Coulson and Sllberman, 1959, i960; Evans, Glaser, and 
Homme, 1959; Shay, 1961; Hamilton and Porteus, 1965.) 

It was decided therefore to use large step programing as a means of 
providing a rapid overview or review of the material to be taught. 

Evaluation of the student's knowledge 1s bound up with two desirable but 
mutually exclusive characteristics. One would like to have detailed and 
reliable Information concerning the student's prior knowledge. Also, since 
the Information 1s to be used to save Instruction time, 1t 1s necessary that 
the evaluation of the student's knowledge n>A. be so time consuming that 
saving 1n Instructional time 1s negative. If the repetitive evaluation of the 
student's knowledge 1s considered within the framework of Information ex- 
change, an alternative to making the evaluation time longer 1s to enrich the 
flow of Information within a given Interval. Other things being equal, 
multiple choice questions take less time than do constructed response questions, 
but multiple choice quest?ons may not provide as much Information as do 
constructed response questions. However, multiple choice questions with care- 
fully constructed choices, coupled with a modified form of responding, may 
provide a rich flow of Information without sacrificing the t1me-sav1ng 
characteristic of multiple choice tests. 

Consider what a student does when confronted with a multiple choice 
question. He considers the choices given with the knowledge that one of the 
choices 1s the correct answer. This automatically rules out many Incorrect 
answers the student might have given to a constructed response form of the 
same question. The student may next rule out one or more of the chokes as 
definitely Incorrect. From the remaining choices, the student selects the 
most likely answer. Any uncertainty on the part of the student as to which 
answer 1s correct 1s lost 1n this form of responding. All the Information 1s 
binary, the student 1s either right or wrong. However, 1f Instead of choosing 
one correct answer to a multiple choice question the student 1s Instructed to 
state his subjective probability (De Flnettl, 1965) or degree of certainty for 
each choice, the amount of Information which can be obtained from a multiple 
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choice Item can be Increased. (See Shuford, Albert, and Massenglll [1966] on 
this mode of response.) For example, suppose a student 1s presented with a 
multiple choice Item for which ho 1s certain that choices C and 0 are Incor- 
rect, and he Is 7035 sure choice A 1s correct, yet he has a 30% degree of belief 
that choice B may be correct. In the normal mode of responding the student 
would choose answer A and the fact that he had some belief 1n answer 6 would be 
lost. However, 1f the student were responding by stating his subjective 
probability for euch response, this uncertainty could be expressed. This form 
of responding which allows the S to express his present state of knowledge for 
a test Item was employed In embedded tests of the Ss' prior knowledge of the 
concepts on which Instruction was to be presented. In order to process 
responses 1n this form on the I8M 1410 system 1t was necessary to write a 
rather lengthy subroutine. For a description of this subroutine, see pages 56, 
56, arid 57 of this report. 

Typical small step, high response rate programing was employed to In- 
struct students whose performance on the embedded tests Indicated they were 1n 
need of Instruction or remediation on some concept or subconcept. 

A description of tho Integration of the three techniques 1n the 
experimental program 1s presented 1n the section on materials and 1s shown in 
Figure 1 of this report. 

Method 

Subjects 

S1xty-f1ve volunteers from an Introductory educational psychology course 
at The Pennsylvania State University served as subjects. 

Materials 

Two CAI programs were used. One program consisted of a section of a 
modem mathematics (MMS) course developed at the Computer Assisted Instruc- 
tion Laboratory of The Pennsylvania State University. 1 The section selected 



*The Writers would like to thank Professor C. Alan Rtedesel and Professor 
Marilyn Suydam, who developed the modern mathematics program. 
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consisted of instruction on base conversion of number systems with bases other 
than base ten* This program Is essentially linear with remedial loops. The 
course from which this section was taken has been rather extensively evaluated 
and revised on the basis of student performance records , and there 1 s ev1 dence 
that It produces satisfactory criterion performance. The minimum number of 
responses possible to complete this program segment was 61, the maximum 
number of frames was 107 with repeated responding possible on some frames. 

The second program (EXS) consisted of Instruction on the same concepts 
presented In the first program. In designing this program an effort was made 
to Implement a strategy which would be responsive to individual differences In 
amount of prior knowledge of the subject matter. Specifically, the program 
had the following characteristics: 1) large steps, 2) embedded tests of 
student performance, and 3) "ski p ahead" and remedial branching. The ml nlmum 
number of responses possible to complete this program was 19, the maximum 
number of frames was 81. 

There were two (large step) blocks of Instructional material 1n the 
program, one dee? Ing with the place value system and the other with algorithms 
for converting from one number base to another (see Hgure 1). 

Three tests were embedded 1n the program. The purposes of the tests were 
to assess the student's performance, diagnose specific problems the student 
might be having and thereby provide Information on the basis of which the 
student was either branched ahead or branched to remedial material . These 
tests consisted of multiple choice questions to which the student responded 
by stating his subjective probability or degree of belief for each of the 
choices. 

Procedure 

Subjects were randomly assigned to each of the two programs, 33 Ss were 
assigned to EXS and 32 to MMS. Subjects were run Individually on IBM 1060 
terminals. Each subject was administered a 22-1 tern pretest Immediately before 
going on the program, a 22-1 tern post test, an'/, an 11-1 tern transfer test Imme- 
diately following his completion of the program, and a 22-1 tern retention test 
one week later. The pre-, post-, and retention tests specifically Included 
conversions from one base to another, which were taught 1n the program. The 
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transfer test required the Ss to perforin addition and subtraction in bases 
other than base ten. A11 four tests required constructed responses. The 
reliability estimates for the respective tests by Kuder-Rlchardson Formula 20 
are as follows: pretest, .93 (from a prior study; Wodtke et a1.» 1957); post- 
test, <93; transfer test, .93; retention test, .96. 

Results 

Students who received non-zero scores on the pretest or who had received 
prior Instruction on numbers systems other than base ten were categorized 
as having prior knowledge (PK) of the content. Students who scored zero on 
the pretest and had received no prior Instruction on numbers systems other 
than base ten were categorized as having no prior knowledge (NPK) of the 
content. Performance data, consisting of posttest, transfer test, and 
retention test scores along with Instructional time, were analyzed within a 
2 x 2 analysis of variance design. One factor consisted of program (MMS or 
LXS), the other of extent of prior knowledge (PK or NPK). 

The results of the analysis of variance of posttest scores for the PK 
and NPK groups by program are shown 1n Table 1. Neither of the main efforts 
were significant; however, there was a marginally significant Interaction 
between extent of prior knowledge and program taken (p <,10). The means, 
standard errors, and n for this analysis are given 1n Table 2. The marginally 
significant Interaction 1s shown 1n Figure 2. 

Table 1 

Analysis of Variance of Posttest Scores 
for PK and NPK Groups by Program 

Source d.f. M.S. F P 



Program 1 23.01 <1.0 

Prior Knowledge 1 72.38 1.84 >.10 

Interaction 1 126.11 3.21 <.10 

Residual 61 39.25 
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Table 2 

Means, Standard Errors, and n for the Posttest 
by Program and Extent of Prior Knowledge 



Group 


EXS 


MMS 






x * 17.38 




x • 15.73 


PK 




1.74 n ■ 13 




1.89 n -11 






x ■ 12.30 




x- 16.43 


NPK 




1.40 n ■ 20 


£n - 


1.37 n ■ 21 



Posttest 
Score 



201 

15 

10 
5 " 
0- 



EXS 

— MMS 



— _ < — — 

PMor 
Knowledge 



4- 



No Prior 
Knowledge 



Fig. 2. Plot of the means of the posttest scores for PK ant NPK Ss on MMS and EXS, 
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The results of the analysis of variance of retention test scores for the 
PK and NPK groups by program are shown In Table 3. The results of this 
analysis are very similar to the results of the analysis of posttest perfor- 
mance. The main effect for programs was not significant. The main effect 
for extent of prior knowledge was significant (P <. 05)} however, as on the 
posttest, there was a marginally significant interaction between extent of 
prior knowledge and program taken (P <.10). The means, standard errors, and n 
for this analysis are given In Table 4. The Interaction 1s plotted In Figure 3. 

The results of the analysis of variance for the transfer test scores are 
shown 1n Table 5. There were no significant differences 1n retention test 
performance. 

Table 3 

Analysis of Variance of Retention Test Scores 
for PK and NPK Groups by Program 



Source 


d.f, 


M.S. 


F 


P 


Program 


1 


41.80 


1.01 


>.10 


Prior Knowledge 


1 


234.64 


5.69 


<.05 


Interaction 


1 


134.51 




<.10 


Residual 


58 


41.25 
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Table 4 

Means, Standard Errors, and n for the Retention Test 
by Program and Extent of Prior Knowledge 



Groups 


EXS 








x « 19.17 




X » 


17.80 


PK 


(5*. 


1.85 n * 12 




2.03 


n ■ 10 






x ■ 12.00 




x * 


16,81 


NPK 




1.47 n - 19 




1.40 


n ■ 21 



Retention 

Test 

Score 



20/ 
15- 

)<y- 

5- 

o- 



~ EXS 
— WIS 



-f 



Prior 

Knowl edge 



No Prior 
Knowledge 



Fig, 3. Plot of the means of the retention score for PK and NPK Ss of HMS and EXS. 
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Table 5 

Analysis of Variance of Transfer Test Scores 
for PK and NPK Groups by Program 



Source 


d.f. 


M.S. 


F 


P 


Program 


1 


4.23 


<'(.0 




Prior Knowledge 


1 


25.49 


1.67 


>.10 


Interaction 


1 


2.24 


<1.0 




Hos 1 dual 


57 


15.23 







The analysis of variance results for Instructional time are presented 1n 
Table 6. There was a significant main effect for programs (P <.05), The main 
effect for extent of prior knowledge as well as the interaction was non- 
significant. The means, standard errors, and n for this analysis are given In 
Table 7. Ths means are plotted 1n Figure 4. 

Table 6 

Analysis of Variance of Instructional Time 
for PK tnd NPK Groups by Program 



Source 



Program 

Prior Knowledge 

Interaction 

Residual 



d.f. 



M.S. 



1 
1 
1 

61 



4018.89 
2800.07 
982.20 
952.99 



4.22 
2.94 
1.03 



<.05 
>.10 
>.10 
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Table 7 

Mean Instructional Time 1n Minutes, Standard Error, and n 
by Program and Extent of Prior Knowledge 



Group 


EXS 


MMS 


PK 


X - 69.97 
(J? - 8.56 n - 13 


x « 94.38 
$x"« 9.31 n - 10 


NPK 


X » 91.68 
6* - 6.90 n ■ 20 


x » 99.94 
-6.74 n - 21 



100- 
90 ■ 
80 
70+ 
60 



WTor" No Prior 

Knowledge Knowledge 



EXS 

MMS 



Fig. 4. Plot of mean Instructional time for PK and NPK Ss on MMS and EXS. 
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Because of the marginally significant Interaction between program taken 
and extent of prior knowledge for posttest and retention test data, the simple 
main effects were calculated. From analysis of the simple main effects, the 
following pattern emerges. PK Ss on EXS do significantly better than NPK Ss 
(P <.05) for the posttest, and (P <.01) for the retention test. Ss on MMS 
having PK do not do significantly better than Ss having NPK (F <1 .0). 

The simple main effects of PK - NPK across MMS - EXS are as follows: for 
Ss having PK there were no significant differences 1n performance on posttest 
or retention test attributable to the Instructional programs (Fs <1.0). Ss 
having NPK did significantly better on MMS on both posttest (P <1.0) and 
retention test (P <.10). 

The EXS program seems to nave capitalized on the knowledge which Ss had 
prior to Instruction. The finding of no significant differences between MMS 
and EXS on posttest, transfer test, and retention test, coupled with the time 
saving for EXS, suggests that students with prior knowledge would benefit by 
having Instruction on EXS rather than MMS. For NPK Ss the lower posttest and 
retention test scores on EXS seem to call for MMS for these Ss 1n spite of 
the time saving on EXS. 

Discussion and Conclusions 

The procedure reported here seems to provide a means of adapting to 
extent of prior knowledge which results In considerable time saving with no 
decrees* 1n criterion performance. The results of this study also suggest 
that neither of the programs could be recommended for all students 1f they 
vary widely 1n extent of prior knowledge. Perhaps parallel programing 
employing the formats of both the programs with a branching procedure for 
switching students from one program to the other my provide the benefits of 
both. 

The procedure employed 1n EXS may have somewhat limited application 1n 
terms of the Instructional content. The content 1n this study was such that a 
student's prior knowledge of a concept could be evaluated. For content 
consisting of more or less discrete units of Information, evaluation of prior 
knowledge by the method employed here may not be feasible. 
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NUMERICAL AND VERBAL APTITUDE TESTS 
ADMINISTERED AT THE CAI STUDENT STATION 

Joseph L, French and John Tardibuono 1 

Each year the youth of our country spend more time in psycho-educational 
testing than the previous year. Not only are more and more tests being used, 
but the length of testing time has Increased as 1t has become apparent that 
short conventional tests are less reliable than long ones. Probably no 
single school -related activity 1s more distasteful to the average student 
than the experience of being tested. Even though a major part of this negative 
attitude 1s the result of anxiety aroused by the importance of the test, 
there remains a substantial portion which becomes associated with the 
experience Itself. The picture of the anxiety-aroused student dropping papers 
and pencils, overlooking a page of Items, looking at another student's paper, 
miscalculating his time, and answering numerous questions which are either too 
easy or toe difficult, 1s a familiar one to most people. It 1s obvious that 
more of tha above-mentioned problems can be attributed to both the physical 
aspects Of the test environment and to the properties of the test Itself. 

The utilization of computer-assisted Instruction (CAI) equipment for 
testing can do much to alleviate many of these problems* First of all, the 
CAI terminal provides a well -control led environment with minimal distraction. 
Secondly, the branching capabilities of the computer appear well suited to 
the construction of a test with a format 1n which 1) each student attempts 
only those Items which are most appropriate to his particular ability level, 
and 2) the time permitted for each, Item 1s the same for everyone. It was 
hypothesized that a program could be developed for such a test which would 
take the subject in relatively large steps to the area of his threshold of 
understanding, and 1n that area move the subject 1n small steps to the point 
where he 1s unable to answer Items correctly. 8y minimizing the very easy 
and very difficult Items and maximizing the number of Items at one's thresh- 
hold of understanding, a highly reliable and valid test should result. In 



'Special acknowledgement 1s due Mrs. Jacqueline Sal lade, graduate 
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order for the procedure to be made operational it was necessary to assume that 
1) Items could be arranged 1n increasing order of difficulty and 2) the 
subject's score on such a test would define a point below which all eliminated 
Items would be passed and above which all items would be failed. 

The present project was an attempt to produce a test which, when compared 
with traditional tests, will 1) have fewer Items for each examinee, 2) require 
less examiner time, 3) be more reliable, 4) be just as valid, and 5) be viewed 
with favor by examinees. 

Because the building of a test suitable for administration via computer 
Introduces many problems which are conceptually different from those Involved 
In traditional testing, 1t seems more reasonable, at this time, to specify 
goals or objectives rather than formal hypotheses. In constructing a Computer 
Assisted Test (COMPAT) four goals were formulated. 7he purpose of this study 
was to arrange tests using multiple choice, numerical and verbal items, In a 
program so that: 

(1) the average number of items attempted by each student will be 
significantly less than the number of Items traditionally required by a test 
of similar content (1n this case, the Henmon-Nelson Tests of Mental Ability) ;^ 

(2) the test will take significantly less time to administer than the 
traditional test; 

(3) Indices of reliability obtained by means of internal consistency 
formulas (KR 20) will be higher for COMPAT than for those reported 1n the 
manual for the Henmon-Nelson tests* and 

(4) there will be a significantly high correlation between each COMPAT 
and the Henmon-Nelson Test, and between each COMPAT and student performance 
In other academic activities. 

The various steps involved 1n this report will be described as follows: 
1) description of the equipment, 2) selection of Items and final format of 
COMPAT, 3) format of the COMPAT program, 4) sample characteristics, 5) 
procedure, results, and graphic description, 6) conversion to the IBM 1500 
system, and 7) plans for future research. 
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Equipment 

The basic equipment consisted of two units: a 1410 IBM computer located 
a few blocks from the CA! Laboratory and an IBM 1050 terminal composed of a 
standard Selectric typewriter keyboard and, under program control, an 80 slide 
random-access carousel projector with a rear Image screen. Four terminals were 
available for subjects 

Development of the Tests 

The process of Item selection necessary to Implement the above-mentioned 
testing conditions Included several stages. In order to minimize the Initial 
problems of Item validity and level of Item difficulty, permission was 
obtained from the Houghton Mifflin Company, publishers of the Henmon- Nelson 
Tests of Mental Ability , to use many of the Henmon-Nelson items for this 
study. The Henmon-Nelson items are predominantly verbal and numerical and 
appear 1n convention^ spiral -omnibus format in two parallel forms at each 
academic level. To obtain a large pool of Items for separate numerical and 
verbal tests suitable for a Vocational Technical school population, Items were 
selected from the High School Form A, High School Form B, College Level Form 
A, and College Level Form B tests. The result was a pool of more than 200 
verbal Items and about 100 numerical items. Although each of the four tests 
presented the Items in order of Increasing difficulty, the amount of overlap 
between the high school and college levels was not known. The four forms 
were administered to approximately 100 high school students from several 
communities to further determine the difficulty level of each Item. 

The next step involved a cross check of the order of difficulty. The 
verbal items were rearranged 1n agreement with the obtained difficulty levels 
and administered to 53 Penn State undergraduates. The same procedures were 
followed with the numerical questions. Some additional re ordering of Items 
followed. 

Utilization of the IBM 1410 computer system Imposed a limitation of 80 
Items for presentation via a carcusel slide projector ; a limitation neces- 
sitated by virtue of the study which preempted the loading of additional trays. 
The complexity of the branching program served to limit the number of Items as 
well. 

o 
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From the above-mentioned Henmrn-NeUon Hems, seven tests were generated/ 
They were as follows: 
1* Pretest 

"Approximately every eighth Item from the pool of 229 verbal Items 
was extracted and used to construct a 2 5- Item pretest. It was 
orglnally planned to administer this test via the computer, but 
preliminary investigations Indicated that such a procedure In- 
volved Inefficient use of machine time. This test was subsequently 
administered In paper and pencil form to determine which of tests 
Verbal A, B» C» or 0 should be administered. 

2. Verbal A-B-C-0 

" From the 204 verbal Items remaining after 25 hari been used 1n the 
pretest, four 80- Item tests were formulated; Test A contained the 
easiest 80 1 terns from the ordered pool ; Test B 1 ncl uded t:.e last 
38 Items from Test A plus 41 of the next easiest Items; Test C was 
composed of the last 38 questions from Test B plus the next 42 Items; 
Test D consisted of the last 39 Items from Test C and the rest of 
the 204 questions. Which 80-1tem test was administered depended on 
the score obtained on the pretest. 

3. Verbal 80 (V-80) 

In order to test the possibility that fewer items, with greater 
difficulty Intervals would yield results equivalent to use of the 
above tests, a third verbal test was developed. In this case, every 
fourth Item (in difficulty level) was used to construct an 80- Item 
verbal test. Items in the V-80 test also appeared on the four 
tests described in 2 above. 

4. Numerical 80 (N-80) 

From the pool of numerical Items, 80 of the most difficult ones 
were used to form the numerical test. 



format of the COMPAT Program 

The program used to present COMPAT was written 1n CoursewrUer author 
language. The complexity of the program Is reflected In the fact that 1t 
Includes about 4425 statements and required 11 hours to compile. However, 
the branching strategies within a single program accommodated six different 
tests and required merely the selective use of a specific tray. The examiner 
Installed the tray with one of the six tests and then signed the subject onto 
the system with a pre-reg1stered student number. In order to change tests 1t 
was necessary both to sign the student onto the system with a new student 
number, and to change slide trays. The sequences of Hem presentation were 
as follows: 
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Each subject started with the eighth Item and was presented every eighth 
Item up to question 60, provided no mistakes were made, at which point he was 
assigned a score of 80 and signed off. However, most students are not able to 
answer all questions correctly. When an error Is made, the branching capabil- 
ities of the computer m utilized. As soon as the first error was made, the 
system backed up five lams and if the student correctly answered that Item, 
the program moved ahead two Items at a time; If a second error was made, the 
system again backed up five Items and then proceeded forward, one question at 
a time. The system continued In that fashion until the subject missed four 
out of the last seven items presented. A score was then assigned based on the 
highest Item nuitoer answered correctly minus the number of items attempted but 
missed below that point. If at any point during the test, the sequence reached 
an Item which had been previously presented, the advent Hem was presented, 
depending on the direction In which the program hac t-:<in moving orglnally. 
For example, 1f a subject had failed one or more items, and if the program was 
now moving forward and reached Item S4 which had already been presented, Hem 
55 would be given. If the subject misses 59, the next Item should be 64; but 
>- If 54 has been attempted, then Hem 53 or the highest Item not attempted below 
54 would be given. From Hem 54, the system would back up one Hem at a time 
until an unattempted Hem was found. Each Hem not answered In 60 seconds was 
considered Incorrect, but all the Items remained on the screen until an answer 
was given. The decision to use a 60 -second time limit was based on pilot 
studies. (Further research Is needed on time 11ml ts .) Examples of actual pat- 
terns of response can be found on page 26. 

The Sample 

One of the practical limitations encountered In the first phase of this 
project was the availability of subjects of the appropriate age and educational 
level willing to take time for research projects. Since this project focuses 
on the feasibility of developing short, highly reliable tests, vocational - 
technical school subjects were not sought. The 73 subjects used In this 
reporting period Included high school juniors and seniors, housewives, and 
undergraduate and graduate students from Penn State. It should be noted that 
80 subjects took the various tests. However, systems difficulties and 
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scheduling problems for subjects reduced the number of subjects for various 
computations and analysis. Good data were available for all tests for 67 
Subjects . 

In Table 1 can be *ound data from the standardization -of the College 
Level forms of the Henmon-Helson and for the subjects used ;n COMPAT studies 
responding to Form B. It becomes apparent from Table 1 that In each instance 
the sample for this project obtained higher scores and exhibited greater 
variability than the comparison sampUs. As can be seen 1n Table 2, the 
correlations between the numerical and verbal and total test scores are re- 
markably similar for a random sample of 200 subjects from the Henmon-Nelson 
College Level standardization population and for COMPAT when used with the 
sample described. 

Table 1 

Comparison of Verbal, Quantitative and Total Scores 
from Henmon-Nelson Standardization Oata 
witn Scores From the Project Sample on 
College Level Form B 



Henmon-Nelson Samplo of College Freshman 4 


Project Sample 




H 


• 148 


N 


- 95 


N » 73 




Mean 


S.D. 


Mean 


S.D. 


Mean S.D. 


Q 


19.62 


5.89 


19.36 


6.34 


24.08 8.65 


V 


28.86 


10.47 


27.57 


10.17 


38.59 12.74 


T 


48.48 


13.60 


46.93 


14.24 


62,60 19.39 



a The Henmon-Nelson Tests of Mental Ability Examiner's Manual; 1961, p. 15. 



Table 2 



Interrelations Between Quantitative and 
Verbal and Total Raw Scores 







Henmon-Nelson 




COMPAT 








N * 100 




N » 73 






V 


T 


V 




T 


Q 


.587 


.827 


.66 




.87 






,941 






.94 



Procedure 

Each subject took five tests: 1) the Henmon-Nelson College Level* Form 

B, 2) Verbal 80, 3) Numerical 80, 4) Pretest, and 5) Verbal A, B, C, or 0. 

The directions for the Henmon-Nelson are contained in the test booklet which 

each subject read himself. For the COMPAT teVts, each S was shown how to 

operate the terminal, how to respond, and how to correct a mistake. He was 

Instructed as follows: 

You will be shown questions, either verbal or mathematical 
on the screen at your left. As soon as you have the answer, type 
the number Indicating that answer, then press EOB; the next 
question will then be shown. If you w1sn to change an answer 
before you press EOB, follow the procedure previously described. 
Remember once you press EOB you will not be able to change your 
response. There is a time limit for each question; but 1f you 
work steadily, you need not rush. 1 will be 1n the next room 1f 
any complications arise. Any questions? You are now ready to 
begin. 

Due to the length of time necessary to complete all five tests, most of the 
subjects returned for a second session. The availability of computer time 
as well as consideration for the subject's schedule made exact Intervals 
between sessions Impractical. Each subject was paid a flat rate of $3, 
unless system failures necessitated returning for a third session 1n which 
case payment was made at the rate of $1 .25 per hour. 

As each S entered the terminal room, he was given either the Henmon- 
Nelson College Level, Fom B, the pretest, or assigned to one of the computer- 
administered tests. Oue to the varying lengths of time subjects spent on each 
o ; the tests, it was not practical to randomize the order of the tests. 
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Post-facto examination revealed the order of administration described 1n 
Table 3. The effect of order was not determined 1n the present Investigation, 
(Further studies will be designed to account for the effect of order.) 

Table 3 

Number of Subjects and 
Order of Test Administration 





First 


Second 


Third 


Fourth 


Fifth 


Verbal 80 


18 


41 


4 


4 


9 


Numerical 80 


16 


11 


40 


3 


6 


Pretest 


4 


6 


13 


53 


0 


Verbal A-B-C-D 


0 


4 


9 


7 


56 


Henmon-Nelson B 


38 


14 


10 


9 


5 



It should be noted that the method of Item selection. Involving replica- 
tion of certain Items across the different tests, complicates the Interpreta- 
tion of results. The responses to questions which appeared twice were 
relatively stable, as can be seen In Table 4. Mention should be made of the 
relatively high percentage (22) of Items answered correctly on the Henmon- * 
Nelson Form 8 yet missed on the COMPAT version. One reason for this occurrence 
may be that many of those Items wore scored as Incorrect because Che 60-second 
time limit had been exceeded. 

The above observation becomes crucial 1n v'iew of the actual Influence of 
the time factor 1n the prosent test. Inspection of the data revealed that on 
the Verbal test, 71 of the 117 overtime responses were correct; furthermore, 
on the Numerical test, out of 333 "t1med-out" responses, 235 were actually 
correct. 
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Table 4 

Relationship Between Responses to Items Which 
Appeared on Both the Henmon-Nelson College 
Level, Form B, and Two COMPAT Tests 



No, of 
Hems ap- 
pearing 
Test twice 



N-80 
V-80 



18 
18 



No, Re- 
sponses 
made to 
those 
Hems 



Per cent 
Correct 
on both 



Per cent 
Wrong on 
both 



383 
478 



47 
62 



21 
17 



Per cent 
Correct 
on HN-B 
but Wrong 
Oh CAI 



22 
7 



Per cent 
Wrong on 
HN-B but 
Correct 
on CAI 



8 
21 



Total 
Per cent 
Hatch 



68 
79 



Results 

To achieve goals 1 and 2 required the administration of relatively few 
Hems In a relatively short period of time when compared with the administra- 
tion of the conventional format of the Henmon-Nelson. Table 5 reflects the 
data relating to these two goals. In conventional 'cwat the Henmon-Nelson 
test provides 100 Hems and allows no more than 40 mlutes or working time. 
While not all subjects attempt the 100 Hems, H 1s assumed that as subjects 
* un into uMiituit items they scan the remaining Hems 1n hopes that they can 
find some which they can answer. With the COMPAT procedures, the mean number 
Of Items attempted varied front 23 to 30 «snd the mean number of minutes varied 
from 21 to 31. Relatively large standard deviations can be observed for both 
number of Hems attempted and time spent on the test. It can be seen In 
Table 5 that the objectives of construe* *jv? a test with fewer administered 
Hems than on the criterion test was resllied. wnen the numerical test and 
verbal test were added together, a mean of 57 Hems was obtained. 

The time Indicated 1n Table 6 was obtained by subtracting "sign on" time 
from "sign off" time. The time figures Include system delays and program 
malfunctions. By analysis of the response latencies of a random sample of 25 
subjects, It was learned that the time to complete the test Is decreased 
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significantly when only response latencies to each Item are considered. 
(Translation of Compat to the IBM 1500 system, as described later in the 
report, 1s expected to decrease testing time due to system delay.) 

Table 5 

Means and Standard Deviations of Items Attempted 
on Compat Tests: Compared to Number of Items and 
Time Required for Hennwn-Nelson, College Level, Form B 



HN-B Verbal 80 Numerical Verbal C Verbal 0 



Mean No. of Items 
Attempted 


100 


29.5 


26.5 


22.9 


24.5 


S.O. 




20.8 


18.6 


19.2 




Mean Time on Test 
(In Minutes) 


40 


30.0 


35.7 


20.5 


24.9 


S.O. 




24.3 


24.9 


14.8 


18.5 


N 


m 


67 


67 


17 


41 



Goal 3 pertains to reliability. At this stage of development with these 
tests, test-retest procedures over short periods of time did not seem ap- 
propriate. Statistical formulas requiring that the same Items be presented to 
each subject were Inappropriate also since the same pattern of Hems was 
presented to few subjects. Since great care was exercised 1n the selection of 
Items arranged 1n order of difficulty, 1t was assumed that all Items which were 
not administered and which were below the score would have been passed. It was 
further assumed that all Hems which were not administered and which were above 
the subject's score would have been failed. Since these assumptions are 
Implicit in the computation of the score, H 1s reasonable to assume them 
also 1n the computation of reliability coefficients. (Such procedure 1s 
followed 1n the administration of such tests as the Stanford-B1net.) A 
demonstration of the patterns of actual response can be found 1n Figure 1. 
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No. S35 




S69 


S10 


S13 


S80 


I 






4 






2 






4 






3 












4 












5 






4 






6 






4 






> 






4 






$ 4 


♦ 


♦ 


4 


4 . . 


- ...t. 








4 






10 






<♦> 






il 










12 






4 






13 






4 






14 






0 






IS 






4 






,16 4 


♦ 


♦ 


0 


4 


4. 


17 






0 






ia 






0 






19 


♦ 




4 










♦ 


4 






a 


♦ 










22 




4 


4 






23 


4 


4 


4 






24 4 


0 


4 


A 


+ 


* 


if 


♦ 


♦ 


(♦> 






26 


♦ 


4 


(♦> 






2? 


♦ 


♦ 


<♦> 






28 


0 


4 








2$ 


♦ 


4 








30 


♦ 


> 








31 


♦ 


4 








32 4 


♦ - 


4 




t 


* 


55 6 


♦ 


4 








34 


4 










3$ 


♦ 










36 


♦ 


4 








3? 


0 


4 








38 0 


0 


4 








19 


0 


4 








40 4 


0 






* 


* 


tl 




4 








42 




4 








43 0 




4 








44 




4 






4 


45 




4 






4 


46 




4 






4 


47 




4 






4 


48 0 




4 




... t 


* 


0 










4 


50 




4 






4 


$1 




4 






4 


52 




4 






4 


53 




4 






4 


54 




4 






4 


55 




4 






4 


56 




4 




4 


«\ 


57 




4 




4 


4 


58 




4 




0 


0 


59 




4 






4 


60 




4 






(♦) 


61 




4 






0 


62 








<♦) 


4 


63 




4 






4 


i* 




4 




4 _ 


*. 






4 






0 


66 




♦ 






4 


67 




4 




(♦) 


4 


68 




4 






4 


69 




+ 






<♦) 


70 




4 






* 


71 










4 


V 




4 






4. 


73 










4 


74 




I 






(♦> 


75 




0 






4 


76 




0 






0 


77 




0 






0 


78 




4 






0 


79 




4 






0 


80 




0 






0 



Score* i 38 



34 



69 



17 



62 



6ft 



4 vorrect response within tl*t 11*1 1 
(4) correct response in exteis of ttn« Unit 
0 incorrect response 



1. A demonstration of the patterns of actual response. 
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The patterns of response for six subjects (In Figure 1) were selected to 
show a variety of responses rather than to depict group tendencies. Subjects 
29, 10, and 80 provide illustrations of the system working as designed. Of 
these, Subject 10 appears to be weakest in measured ability, and Subject 80 
seems to be strongest. The Influence of time in determining a score is 
Indicated by plus signs 1n parentheses. It can be noted that Subject 10 and 
80 correctly responded to four items for which they did not receive credit 
because they exceeded the time limit in making their response. 

Subject 69 answered items 18, 16, 24, and 32 correctly, then failed 
Hems 40 and 35. Item 30 was answered correctly. Eventually, the subject 
tried all Items between 20 and 80 before the test was discontinued. In this 
Instance, little time saving was achieved. 

Subject 35 and 13 present somewhat different problems. Subject 35 
responded to Items 8, 16, 24, 32, and 40 correctly. Then items 48, 43, 38, 
and 33 were failed, and the test was discontinued. With this pattern of 
response, one is not sure whether the score of 38 1s a reliable one or not. 
It would have been desirable to see more of the student's ability on items 
32 through 48. 

A similar situation can be observed 1n following the pattern of response 
for Subject 13. The subject was correct on the first 8 Items but failed Item 
72. Item 67 was then answered correctly but In more than 60 seconds. Item 
62 was answered correctly, but 1n more than 60 seconds. Although item 57 was 
answered correctly item 58 was failed and the program was discontinued. The 
program 1s based on the assumption that the Individual will, at some point, 
begin to fall items and will then pass and fall a block of items. A discon- 
tinuation after 4 of 7 Items have been failed was selected because in many 
Instances the individual will try a difficult Item and fail 1t, an easy item 
and pass 1t, a difficult Item and fall, an easy item and pass 1t, etc, The 
patterns of Subject 29 and 10 are typical. Confidence can be placed in the 
scores for Subjects 69 and 80 also. 

In Table 7 can be found Kuder-R1chardson Formula 20 reliability coef- 
ficients for the four C0HPAT tests used with enough subjects to warrant 
computation. It should be noted that Kuder-Richardson formulas yield slightly 
higher correlations than other methods of measuring reliability and that the 
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technique of assigning a minus or a plus to unanswered Items will tend to 
Inflate the obtained correlations. The Kuder-Richardson coefficients ranging 
from .977 to .988 compare favorably with the odd-even reliability coefficients 
of .94 and .95 reported in the technical manual for total scores of the college 
level Henmon-Nelson tests. The technical manual reports alternate form 
reliability coefficients of .84, .376, and .887 for Q, V, and total scores. 
Thus, It 1s concluded from the first phase of the reliability study that a 
highly reliable test can be adapted for presentation by a computer. Future 
studies will Investigate other aspects of reliability. 



Table 7 

Means, Standard Deviations and KR20 
Reliability Coefficients for Computerized 
Tests of Mental Ability 



Verbal 80 Numerical Verbal 0 Verbal C Verbal B Verbal A 



Mean 


44.66 


35.56 


54.67 


39.52 


40.5 


16.3 


S.O. 


22.75 


18.14 


22.28 


25.81 


* 


* 


KR20 r 


.986 


,977 


.985 


.988 


* 


* 


N 


69 


71 


40 


18 


7 


3 



*not computed 



The final area of Investigation to be reported here Involves a comparison 
between COMPAT and the conventional Henmon-Nelson test. These data can be 
found 1n Table 8. The relatlbnship between COMPAT and the conventionally 
administered Henmon-Nelson test are not as substantial as those reported 1n 
the Examiner's Manual for alternate forms of the Henmon-Nelson. Future 
studies will deal with some of the Issues raised by these figures and bear 
heavily on goal 4. 

It 1s evident from the data 1n Table 8 that the normative data for the 
Henmon-Nelson can not be used to Interpret COMPAT scores. Norms will need to 
be developed If COMPAT scores for Individuals are to be Interpreted. 
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Table 8 

Correlations Between Scores Obtained on 
the Henmon-Nelson Test, College Level, Form B 
and Three Computerized Tests 



Henmon-Nelson COMPAT 
Total* Verbal* Quant1at1ve* Numerical* Verbal C** Verbal [>*** 



Compat 

Verbal 80 .701 


* 

.760 


.463 


.352 


.577 


.751 


H-N 
Total 


.947 


.861 


.558 


.762 


.574 


Verbal 




.659 


.419 


.758 


.648 


Quantftatlve 






.662 


.548 


.406 


Compat 
Numerical 








.415 


.286 



*N-67, **N-17, ***N-41 

It should be noted that the number of subjects who took COMPAT Verbal -C 
or Verbal-D was dependent on their score on the pretest. Since COMPAT Verbal-A 
and Verbal-B were administered to only 10 subjects, correlational data for 
those tests were not computed. 

Conversion to the 1600 System 

Shortly after the present project became operational for experimental 
research, the IBM 1500 system became available. The 1500 provides several 
distinct advantages over the system used for the collection of data reported 
here: 1) presentation of Hems via a cathode ray tube (CRT) thus eliminating 
the necessity of changing slide trays and the limitation of 80 questions, 2) 
the CRT will black out when time limits have been exceeded, 3) greater flexi- 
bility 1n branching 1s available fn the author language, 4) more terminals are 
available to permit a greater number of subjects to be tested simultaneously, 
and 5) the students will respond by using a light pen (LP) which will 
eliminate some typographical errors. 
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Another distinct advantage Is that the processing capabilities of the IBM 
1500 system Is faster and less t1m* 1s needed to search for Hems when the 
program Is branching because the efficiency of the program 1s Increased by 
using t Buffer to Return Register function which allows a counter to be loaded 
Into a Buffer and a Buffer Into a Return Register. This procedure 1s being 
used 1n conjunction with the Move function to branch to the label located 1n 
the Return Register. The programing described 1n this paragraph replaces the 
use of 80 conditional branch statements with branching on the basis of a counter 
to the next question. 

Also used In tha 1500 system 1s the Buffer to Counter function which 
converts the Buffer Into Counters to store the question number 1f the subjects 
respond with a wrong answer or "time out"; but only 1f that question 1s 
numerically higher than the highest one they have answered correctly at that 
point 1n time does this event occur. When the student has missed four out of 
the last seven questions, this Buffer will be checked to see 1f the number' 
stored 1n the counter within the Buffer 1s less than the highest one correct. 
With this procedure 1t 1s possible to know the number of questions missed 
below the highest one answered correctly. The COMPAT programs have been 
converted for use on the 1500 during this reporting period. 

Future Research Direction 

Now that the programs have been transferred to the 1500 system, 1t 1s 
more feasible to test the level of difficulty of each Item on-Hne. A 
program 1s being written In which each Hem can be presented 1n such a way 
that the current order can be used without branching. It 1s Important to 
have such a check since level of difficulty often changes with mode of pre- 
sentation. To date, levol of difficulty has been obtained by off-line testing. 
The next phase of the project will allow for a study of time spent per Hem. 
The 60-second time 11m1t may be adjusted after reviewing these data. 

Furthermore, during the next phase of the project, validity studies will 
be undertaken. Correlational data will be developed for the various COMPAT 
programs and grade-point average and other aptitude tests available for 
subjects. During this phase of the study 1t will be possible to ascertain 
the feasibility of retaining the long verbal test. So far H appears as 1f 
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the 80-1tem verbal test has adequate reliability. If 1t 1s equal to the 
longer verbal test 1n concurrent validity, the longer test could be abandoned. 

Another study will Involve the development of test-retest reliability 
coefficients. 

A study of reading ability on the CRT 1s under way. Each student 
participating 1n this study will respond to COMPAT V and COM PAT N. The 
relationship of reading ability to the test scores will be obtained. 
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An Experimental Procedure for Course Revision 
Based on Students' Past Performance 

Karl G. 8orman and Donald W. Johnson 

In a previous report (Johnson and Borman, 1967) the authors reported a 
stuaV 1n which they Investigated the time required to complete a program under 
various modes of stimulus presentation (audio static, display, typewriter, and 
slide). No significant differences were demonstrated between the various modes 
of presentation because of the large variances obtained 1n the dependent vari- 
able for the various modes. At that time, 1t was the intent of the authors to 
analyse each subject's performance on each frame In order to determine whether 
or not there were certain frames in the program that contributed a large pro- 
portion of variance to the total within variance group. Table 1 contains the 
means and standard deviations of program completion times from data obtained 
and Included 1n a previous report. (Semi -Annual Progress Report, December 31, 
1967, Report No. R-9, Ex perimentation with Computer-Assisted Instruction 1n 
Technical Education) . 

Table 1 

Mean and Standard Deviations of Program Completion Times 



Group n Mean S.D. 

(in seconds) (In seconds) 



Audio 20 3319.60 1147.65 
Loose-leaf Chart 

Static Display 23 2674.04 665.64 

Typewriter 24 3003.79 1011 .68 

SHde 20 2885.60 782.52 



A Student Records Program (Bahn, 1966) provided the authors with a listing 
of each subject's performance on each frame 1n the program. (Because of a 
programing error, these data were available for cnly three of the four groups; 
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data were not available for the group which received the stimulus material from 
the typewriter. The error was corrected 1n a later experiment.) The listing 
provided the authors with the response entered by the sti-dent, the number of 
attempts on each question, and the response latency for each response. The 
mean number of attempts per student and the mean response latency per student 
were calculated. From these data, the mean number of attempts and the mean 
response latency per group was calculated and plotted on a graph. A sample of 
these graphs Is shown 1n figures 1 and 2. 

Examination of these graphs resulted In the following conclusions: 

1. In general, the group which received Instruction in the audio mode of 
presentation had a tendency to have a higher mean response latency as well as 
a higher number of attempts per frame. 

2. In general, the chart display mode of presentation produced the best 
results; that 1s, tended to have the lowest mean response latencies and the 
lowest mean number of attempts per frame, (although the differences were non- 
significant). 

3. On certain frames, the pattern was reversed, I.e., the group receiving 
Instruction 1n the audio mode had the lowest mean response latency while the 
group which received the Instruction 1n the display mode had the highest mean 
response latency suggesting a frame-by-chart-mode-of-presentatlon Interaction. 

4. Certain frames showed a high mean response latency and a high mean 
number of attempts for all groups which seems to Indicate a difficult question 
regardless of mode of presentation, 

Based on the above Information, the authors examined the frames and the 
student responses In order to determine what was causing the high mean 
response latencies and the high mean number of attempts. In some cases 1t was 
found that the student knew the correct answer and entered a form of the 
correct answer, but the computer would not accept the response because of 
the author's failure to anticipate a particular correct answer 1n programing. 
An effort was made to correct the programing wherever possible. 

In other cases 1t was found that some frames were too long, especially for 
the group receiving the material from audio tape. Information pres&nted during 
the early part of the frame was forgotten by the erid of the frame when the 
subject was required to use the Information. An effort was made to shorten 
the frame, 1n some cases cutting the length of the frame In half*, for example, 
the frame: 




7 9 10 
Frame Number 

F1g. 1. Mean number of attempts per group on each frame. 
All points wlthtn range of rectangle have a value of one. 
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By using this method of working with units, you should 
always be able to calculate units which measure the quantity you 
are trying to find. If you are trying to find a distance and obtain 
a unit of seconds, your answer 1s wrong. If you calculate the proper 
units, this 1s a clue that you are working the problem correctly. 

You will have better success 1n learning how to use units 
1f you actually write each problem on a paper and perform the 
computation. Be sure to cancel units where 1t 1s possible. 

was split Into two frames, each one containing a single paragraph. 

It was found that certain questions were too long to remember, especially 
for a naive subject in the given subject matter. For example, the following 
frame has a high mean number of attempts. 

Each of the following quantities contains a number and 

a unit. Type the words which are the units 1n each of the 
quantities. 

5 miles 2 hours 3 cm 6 grams 

To compensate for this, the last part of the question (6 grams) was 
dropped 1n the course revision. 

Further analysis of mean response latencies and mean number of attempts 
showed that students were not asked to respond to material presented 1n the 
course until 4 or 5 frames had passed. In the revised course, an attempt was 
made to examine the student on each piece of new Information Immediately after 
presentation. In some cases additional drill material was added to the course 
to Insure that the student mastered the concept before proceeding to new 
material based on the concept. 

In restructuring the material the concept was Introduced, an application 
presented, and students were then tested on the application. For example, the 
foil oiHng frame: 

The electric company charges you on the basis of how 
many kw-hr (kilowatt-hours) of electricity you use. 

When units of kilowatts are multiplied by units of hours, 
the unit obtained 1s kw-hr. 

How much electrical energy do you pay for 1f you used a 75 
kw electrical motor for 4 hours? 

was changed to: 
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When units of kilowatts arc multiplied by units of hours, 
the unit obtained 1s kw-hr. 

The electric company charges you on the basis of how many 
kw-hr (kilowatt-hours) of electricity you use. 

How much electrical energy do you pay for 1f you used a 
75 kw electrical motor for 4 hours? 

Upon completion of the course revision, the course was tested on a small 
number of students 1n order to determine the value of the revisions. 

Method 

Materials 

ijpon completion of the revisions, as described, the course, orglnally 
titled "Working With Units," contained 47 frames Including 6 slides common to 
all modes of presentation. The sequence was designed so that all Ss received 
four warm-up frames, each frame presented 1n a different mode of presentation, 
the main purpose of which was to acquaint each Ss with the correct method of 
termln.1 operation. Following the Introductory material , 43 frames of material 
were presented for which data were collected and analyzed. 

In order to provide for a variation In stimulus modes, four versions of 
the course were created. The material from the 37 frames {not Including slide 
material) was presented four ways. One group received this material on audio 
tape to provide the audio mode of presentation. One group received this 
material printed on charts and put together In u booklet, each page of which 
contained the material from one frame of the program. The program Instructed 
the Ss to read a given page. This mode of presentation provided the chart 
display mode of presentation. In a third version of the course, the type modo, 
ths material was typed to the student on the typewriter associated with the I8M 
1050 computer terminal. The fourth group received the Instructional material 
on 2 x 2-Inch photographic slides; the material was Identical to that contained 
on the audio tape, chart display group, and typewriter output. All groups 
received Identical versions of the course and all groups were required to 
answer the questions by typing their answers on the typewriter keyboard at the 
terminal. 

A 20-1tem constructed-response test was created. The test was designed 
to measure factual material as presented 1n the program as well as a subject's 
ability to transfer what he learned to similar problems. For example, 1n 
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addition to dividing meters by seconds, a concept taught In the course, the Ss 
also had to divide ficticious units such as dividing "yens by fuds." Since the 
program was not designed to teach computational skill, 1t was decided to 
score the test only on the basis of whether or not the S had the correct units, 
not whether or not the £ had the right numerical answer. The Kuder-Rtehardson 
Formula 20 reliability of this test was .862* 

Subjects 

The Ss consisted of 33 volunteer upperclassmen majoring In education and 
taking Instructional Media 435 at The Pennsylvania State University during the 
Fall Term, 1967. The Ss were randomly assigned to one of the four experimental 
treatments. The Ss did not have a mathematics or physics background. In ad- 
dition another group consisting of 90 upperclassmen majoring 1n education and 
taking Instructional Media 435 at The Pennsylvania State University during the 
Fall Term, 1967, served as a naive control group and took only the posttest. 

Procedures 

Each S signed on the course and was presented with the instructional 
material and questions based on the material. The S responded to the questions 
by typing his answer on the typewriter keyboard. Feedback material was pre- 
sented by the computer to all Ss via the typewriter. Upon completion of the 
course, Ss were administered a 20 question constructed- response test off- 
line. Total for each question, the response latency for each response, and the 
number of correct responses on the posttest were collected for each subject. 

Findings 

Table 2 shows the metns and standard deviations of the completion times 
(In seconds). Table 3 1s tn analysis of variance summary table for the 
completion times. The F-ratlo was not significant. Table 4 shows the means 
and standird deviations of posttest scores for a;; groups. An analysis of 
variance procedure (Table 5) produced an F-rat1o significant beyond the .001 
level of confidence. Shefte's procedure showed that significant differences 
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Table 2 

Means and Standard Deviations Of 
Completion Times 



Group 


n 


Mean 
(In seconds) 


(1n seconds) 


Audio 


11 


3056.00 


692.75 


Chart Display 


10 


2988.70 


166.11 


Typewriter 


6 


3078.17 


740.76 


Slide 


6 


3062.50 


995.44 






Table 3 






Analysis of Variance For Completion 
Times (In Seconds) 




Source 


d.f. 


Sum of Mean 
Squares Square 


F 


Treatment 


3 


40,888.48 13,629.49 


0.03 


Error 


29 


12,745,466.43 439,498.84 




TOTAL 


86 


12,786,354.94 
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Table 4 

Means and Standard Deviations of Posttest 
Raw Scores (Maximum Possible Score: 20 Points) 



Mode 


n 


Mean 


S.D. 


Audio 


11 


15.45 


1.44 


Chart Display 


10 


14.70 


3.71 


TypewH ter 


6 


13.00 


3.58 


Slide 


6 


16.17 


3.06 


Control 


90 


9.69 


4.56 



Table 5 

Analysis of Variance For Posttest Scores 







Sums of 


Mean 




Source 


d.f. 


Squares 


Square 


F -Ratio 


Treatment 


4 


693.07 


1/3.27 


9.71 (P<.001) 


Error 


118 


2104.95 


17.84 




TOTAL 


122 


2798.02 
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existed between the audio and control groups, the display and control groups, 
and the slide and control groups. The difference in means between the type 
and control groups was not significant. 

Figures 3 and 4 show a segment of the graphs obtained from the revised 
course. Because of revisions, frame numbers 1n figures 3 and 4 are not 
comparable with figures 1 and 2. A general inspection of the graphs shows 
that the majority of the extreme peaks have been tempered, indicating that the 
frames in the course are approaching equal difficulty. It 1s also apparent 
that the type group now has the shortest mean response latencies. This is 
most probably due to the fact that the Ss could read the material as It was 
being typed to them and be ready to answer as soon as the computer gave them 
control of the keyboard. The other groups were forced to read the material 
during the time allotted for responding. Thus, the major differences 1n mean 
response latency are Interpreted as a result of, the programing technique, and 
not of the mode of presentation. This interpretation 1s also reinforced by 
the lack of significance in the obtained analysis of variance on the total 
time to completion data. 

The graph for mean number of attempts per group on each frame also 
Indicates that many of the high peaks were tempered; however, there are 1 or 2 
new peaks to be examined, as well as some smaller peaks. 

Implicat ions 

The main purpose of this experiment was to test a procedure for course 
revision based on past performance. Graphs were prepared to Indicate weak 
sections or frames 1n the course based on high mean response latencies and/or 
high mean numbers of attempts. The graphs also indicated frame-by-mode-of- 
presentatlon Interactions, I.e., indicated frames where one mode of presenta- 
tion was superior or Inferior to the other modes of presentation. It 1s felt 
that by examining these interactions, 1t may be possible to Indicate which 
mode of presentation should be used for which purposes as well as how to best 
use a medium for a specific purpose. For example, it was found that, for the 
given course material, It was necessary to keep the audio messages relatively 
short. 

In revising the course, care was taken to improve those frames which con- 
tributed large proportions of variance to large mean response latencies and 
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Frame Number 

F1g. 3. Mean number of attempts per group on each frame. 

t> M1 points within range of rectangle have a value of one. 
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the high mean numbers of attempts. The revisions would tend to make the course 
more uniform and easier for all groups, since all versions of the course were 
Improved, a condition reflected by the minute mean differences for total time 
to complete the program. A possibility for future research would be to revise 
only the frame for the modality which experienced difficulty, leaving the other 
frames unchanged. Diagnostic revisions may serve to lower within group vari- 
ance and capitalize on the differences that are Inherent 1n the various modes 
of stimulus presentation available with CAI. 

The course material used In this experiment Is the product of a number of 
revisions after It had been carefully written by a subject matter and program- 
ing expert. Even now, It Is far from being optimally efficient and effective 
stimulus material to promote student learning. There are still many peaks and 
valleys that must be accounted and compensated for through revisions or 
branches. The most significant finding to date Is that course development 1s 
a complex, time-consuming process which must be carried out 1n a context where 
student performance data are continually used as a basis for subsequent 
revisions. 
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1500 SYSTEM GEOMETRIC DICTIONARY 
Paul V. Peloquln 

Introduction 

This is a general description of the geometric dictionary used by the 
Computer Assisted Instruction Laboratory at The Pennsylvania State University. 
It 1s hoped that this introduction will enable readers to duplicate and use 
this dictionary on any cathode ray tube (CRT) terminal of the IBM 1500 system. 
Only an elementary knowledge of certain Coursewriter II Instructions will be 
assumed throughout this paper. (The reader is referred to the following 
sections of the Coursewriter II Manual [IBM, 1967]: Part I, pages 2-12; Part 
II* pages 7-15, 58-66.) 

There are at least four advantages 1n using this geometric dictionary. 
First, the geometric dictionary uses less core storage than a graphic set, yet 
there 1s no limit to the number of line drawings which can be made. Because 
the components of a dictionary are small, they become more general and may be 
used 1n many different combinations. By analogy, a graphic set may be equated 
to a vocabulary of 64 words, while the geometric dictionary may be equated to 
an alphabet of 128 letters. The geometric dictionary may be equated and used 
within various courses thereby conserving core storage. Second, the necessity 
of keypunching each and every one of the line drawings dot by dot Is eliminated. 
Third, the geometric dictionary allows the author to construct, on-line, the 
graphics for his course. This ability allows him to Instantly see the line 
drawing as It 1s entered and make necessary changes or corrections. Fourth, 
words and line drawings can be combined without the one column gap necessary 
when using a graphic set. 

There are, of course, some disadvantages as wel 1. The geometric diction- 
ary has been designed for producing moderate and large-sized line drawings. 
Except for some gross shading, such as "blackened" areas and hachures, the 
figures produced with this dictionary have been only line drawings. Some con* 
stramts are placed upon the drawing by the availability and the nature of the 
line segments* This restriction necessitates careful planning, but with the 
help of the IBM 1500 Instructional Display Planning Guide sheets it does not 
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constitute a serious problem. Usually the addition of an extra character Into 
the dictionary or a bit of programing Ingenuity will overcome problems 1n 
producing still or dynamic line drawings. 



General' Description 

The geometric dictionary consists of line segments entered as dictionary 
characters which may be manipulated as such. Throughout this description the 
characters appearing on the terminal keyboard and on the CRT under the system 
dictionary, whether letters, symbols, or numbers, will be referred to as 
'•associated keyboard characters." The characters containing the line segments 
of the geometric dictionary will be referred to as geometric characters, and 
the 1 1 ne parts contal ned within each of the geometric characters w1 1 1 be 
referred to as line segments, whether straight line segments, arcs, or special 
characters. Since we are dealing with dictionary characters, once the 
geometric dictionary has b**n called by a dictionary change the geometric 
dictionary has all the operating characteristics and functions of the system 
dictionary. A particular line segment 1s called and displayed on the screen 
by entering the associated keyboard character in a display text (DT) or 
display text insert (DTI) Instruction, For example: 

DT 12,10///*laaaaa*b*b78 a a a a a 78*b*baaaaa*e 

After a dictionary change (denoted by *1) 1$ made, this instruction will 
display the "a" as a horizontal straight line and the numbers "78 u as the left 
and right half of a small circle respectively. The backspace function (*b) has 
been used to superimpose lines, 

:': : Procedure : 

Sketching 

the procedure Involved In the construction of a line drawing reguifes 
three steps: sketching, coding, and entering. First, a sketch of the line 
drawing is made on an Instructional Display Planning Guide. The sketches 
should be rather simple and should be constructed of those line segments that 
the programer knows are aval Table in the dictionary. On the three pages fol- 
lowing this description of the first step are reference pages Indicating the 
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orientations of the line segments that are presently available to the pro- 
gramer, It is recommended that these reference pages be reproduced in a 
transparent form in order that the programer may overlay the available line 
segments on his sketch for comparison. The reader 1s reminded that each 
geometric character may be used Independently. Thus If the third (and middle) 
character of a 30° line 1s needed, 1t may be used Independently of the other 
four geometric characters which make up the completed line. If the programer 
cannot match his sketch with an available line, he should select the closest 
approximation and revise his sketch accordingly. A limited number of special 
characters may be Inserted Into the geometric dictionary 1f the programer 
finds that their omission seriously handicaps him. Additional flexibility can 
be gained by the use of the keyboard functions such as space, backspace, Index, 
reverse index, and dictionary change. Super1mpos1t1on, offsetting by a half- 
line, the display of text and geometric figures 1n juxtaposition or supeHm- 
positlon, the display of only half of a geometric character, and shading are 
some facilities gained by the use of standard keyboard functions. With a 
little practice, the programer's familiarity with the lines available In the 
dictionary should grow to a point where he will be able to produce sketches 
Which require no lines that are not already 1n the dictionary. In anticipation 
of the second step, the programer may wish to make mental or written notes on 
the line segments he Intends to use 1n constructing the line drawing. 



0 

JC 



Horizontals 

B 



a 



REFERENCE SHEET FOR STRAIGHT LINE SEGMENTS 
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REFERENCE SHEET FOR SPECIAL AND BLANK CHARACTERS 
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Coding 

The second step 1n constructing a figure with the geometric dictionary 
may take place once the sketch 1n the Instructional Display Planning Guide 
contains no lines which are not 1n the geometric dictionary. 

Short Form . The three reference pages preceding this section are used 1n 
the "short form" of encoding. The reader should note that the associated key- 
board characters are written beside, above, or below the geometric characters 
to which they refer. One simply chooses the line segment and geometric 
character he wishes, then encodes the associated keyboard character. 

Long Form. The use of the long form 1s not described here Since simple 
line drawings can easily be constructed with the use of the short form. More 
complex figures requiring knowledge of every lighted dot within the character 
would necessitate the use of the long form. 

The purpose of this abbreviated report 1s to introduce readers to the 1500 
system geometric dictionary being developed at the Penn State Computer Assisted 
Instruction Laboratory. Further Information enabling reproduction and use of 
the dictionary on other IBM 1500 Instructional systems will be made available 
upon final completion and refinement. 
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A PROCESSOR FOR MULTIPLE NUMERIC ENTRIES 
Terry A. Bahn and Bobby R. Brown 

For the sequential testing program (see Prior Knowledge and Individualized 
Instruction 1n this report) 1t was desired that a student be allowed to respond 
to multiple choice questions 1n the embedded tests by stating his subjective 
probability or degree of belief for each of the choices presented. It was 
further desired that the mode of responding not be unduly time consuming or 
unnecessarily restricted 1n the range of format variations accepted. Due to 
the time factor Involved following each student response on the 1410 system, 
It was not desirable to have the student enter his subjective probability for 
each choice separately. If the student was to be allowed to enter his subjec- 
tive probabilities for all choices In a single response, there were two 
possible ways of processing the response 1n which the Information concerning 
the subjective probability for each choice would be preserved. A series or 
"stack" of possible answers could have been provided 1n the program against 
which to compare the student's response. However, because of the number of 
combinations of subjective probabilities possible and the permutations possible 
for each combination, 1t was not practical to employ this procedure. 

The alternative procedure entailed evaluation of each subjective proba- 
bility In the student's response when the Individual probabilities were them- 
selves components 1n a string of probabilities which made up a single response. 
No such capability exists 1n 1410 CoursewHter. The subroutine described here 
provided this capability and was employed 1n the previously referenced 
Instructional program. 

The algorithm for the multiple entry subroutine Is as follows: 

1. The student's response 1s entered in the form: 

xx, xx 

xx, xx, xx, xx 

til! \ i If . \ II I or : \ ■ ■ % ■ ■ ;r r -\' : : : ; ; ■[■;■:■'■ - . 
xx, xx, xx, xx, xx 

where xx 1s any two-digit number within the range 00 to 99 or the three-digit 
number 100. 

2. An edit function deletes extraneous spaces, letters, and special 
character. 
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3. A series of edit functions rounds the numbers 1n the student's 
response to the nearest ten and converts the number 100 to 99. 

4. The response Is now In the form: 

yy. yy 

yy. yy. yy. yy 

or 

yy. yy. yy. yy. yy 

where yy 1s a member of the set of numbers (00, 10, 20, 30, 40, 50, 60, 70, 
80, 90, 99). 

5. Response processing now enters a series of twenty-two (22) to fifty- 
five (55) Initial character function (1c fn) calls depending upon the number 
of entries (2, 4, or 6) and the values of the entries. The Initial character 
function allows one to compare n Initial characters of a response and to 
include "don't care" characters (1n this case $) which will match any single 
character 1n the string of characters which 1s to be matched. 

6. The first set of Initial character functions compares the first 
three (3) characters of the student's response with answers of the form: 

yy 

7. The second set compares the first seven (7) characters of the 
response with answers of the form: 

$$, yy 

8. If only two (2) entries were required, processing passes to step 13. 

9. The third set of functions compares the first eleven (11) characters 
with answers of the form: 

$$» $$ » yy 

10. The fourth set compares the first fifteen (15) characters with 
answers of the form: 

$$. $$» *$» yy 

11 . If only four (4) entries were required* processing passes to step 



12. The fifth set of functions compares the first ninteen (19) char- 
acters with answers of the form: 

$$, $$• $$» $$» yy 

13. Each time an entry was matched, Its value (yy) was placed 1n a 
counter corresponding to its original position 1n the total response, these 
counters are now added together to see if their total Is one hundred (90 to 
110 to allow for rounding error). 

14. The entry with the highest value 1s loaded Into counter six (c6) 
and a switch 1s set to Indicate the original position of this value. 

15. Control 1s returned to the main program. 

With the advent of the I8M 1500 Instructional system and the extract 
integer function (er fn) this same procedure can be implemented with fewer 
statements and greater accuracy. A macro has been written to accomplish this 
task. The coding for this macro (spb 666) 1s contained In Appendix A. 



Appendix A 
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AA 




2 

4* 


Id 




3 


Id 


Q*c2*e 


4 


Id 




s 


Id 






1 vl 




7 

r 


Id 


l/f •> 1 C 




Id 


G><;2*e 


Q 


Id 
1 vi 




1 v 


Id 




1 1 


Id 
1 u 


5 v c 


1 1 


fn 
i II 


C 1 fV\Jf 1 f v 1 c 


13 


fn 
i ii 


S2\ f\)\Jf Cf\+C X3 




fn 

in 


©if ouf of co*t? 


IK 


vn 




1 A 
1 0 


fh 

in 


e1/b0?5/c5*e 


17 


Id 


C 1 fWo 


18 


ad 


c2/c6*e 


19 


ad 


c3/c6*e 


20 


ad 


c4/c6*e 


21 


ad 


c5?c6*e 


22 


br 


#01##3/c6/1/100*e 


23 


br 


#02##3/c6/g/100*e 


24 


Id 


d/c6*e 


26 


or 


#03M3/c2/1e/c6*e 


26 


Id 


c2/c6 f e 


#03W*e 



1 br *04M4/c3>»1e/c6*e 

2 Id c3/c6*e 
#04M3*e 

1 br #05M3/c4;Me/c6*e 

2 Id c4?c6*e 
#05##3*e 

1 br #06##3/c6/1e/c6*e 

2 Id c5/c6*e 
#061 #3*6 

1 br #07##3)Wne?c6*e 

2 Id 1j»$1*e 
#07##3*e 

1 br #08##3/c2/nefr6*e 

2 Id 1/s2*e 

1 br #09##3?c3/ne/c6*e 

2 If Vs3*e 
#09##3*6 

2 % ijjf 4 ^^ 6 



60 



1 br #H##3/c5/ne/c6*e 

2 Id Vs5*e 

3 br #UM3*e 
#01113*6 

1 it 28 ,5/4, 28/40, 0/The sum of your answers should not*c*ibe LESS than 100. 
Try again. *e 
#02M3*e 

1 dt 28,5/4, 28/40, 0/The sum of your answers should not*c*1be MORE than 100. 
Try again. *e 

2 br re*e 
#HM3*e 

1 en *e 



